Repository navigation
openvmm: let platform snapshots activate vCPUs at restore - #801
Merged
Pedro Henrique Penna (ppenna) merged 5 commits intoOct 25, 2026
Merged
Pedro Henrique Penna (ppenna) merged 5 commits into
Pedro Henrique Penna (ppenna) merged 5 commits into
Conversation
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-platform snapshot and authentication changes require human review of complete exact-head CI evidence.
No open issues 🎉
What changed in this PR
Addresses #796 by updating the OpenVMM pin to support restore-time vCPU activation for platform snapshots and correct authentication timeout handling during gated restores. Adds NVX coverage and measurement tools on top of #797.
Changes:
- Adds a default microVM scenario covering CPU activation, platform repair, invalid targets, and snapshot integrity.
- Adds an explicit platform-snapshot benchmark for 1/2/4/8 online vCPUs, reporting latency, peak memory, and optional lifecycle phases.
- Rebalances CI shards and documents the new behavior and upstream changes.
| File | Description |
|---|---|
| scripts/test_nvx_tools.py | Tests benchmark arguments, capture reuse, targets, and metadata. |
| scripts/test_microvm_tests.py | Tests scenario execution, repair checks, VP bindings, and snapshot integrity. |
| scripts/nvx_tools/microvm_tests.py | Implements the platform restore-processor scenario. |
| scripts/nvx_tools/microvm_shards.json | Adds the scenario and rebalances backend shards. |
| scripts/nvx_tools/benchmark.py | Implements and registers the platform vCPU restore benchmark. |
| doc/usage.md | Documents scenario restrictions, reporting, and suite selection. |
| doc/run.md | Clarifies platform snapshot CPU activation support. |
| doc/openvmm-upstream-roadmap.md | Records the new pin and upstream fixup mapping. |
| doc/ci.md | Describes the new scenario’s correctness checks. |
| doc/benchmarks.md | Documents benchmark usage and inherited Windows capture behavior. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/fix-issue-796-implement-tests-benchmarks
branch
from
October 23, 2026 20:19
2f4bba1 to
f593442
Compare
This was referenced Oct 24, 2026
Pedro Henrique Penna (ppenna)
requested a review
from Enrique Saurez (esaurez)
October 25, 2026 01:51
Pin microsoft/openvmm#4735 at 2b681564dd63483f2f9c9f9bba3b98a4c00e9680, two commits on 8a0d945ed, the previous pin, which is the head of feature/microvm/dev since microsoft/openvmm#4734 merged it. The first commit lets a platform snapshot opt into restore-time vCPU activation. Restore-time memory expansion already worked on every tier, but a platform capture booted with maxcpus=B failed to publish, because the platform tier's command-line allowlist did not accept maxcpus=, and one captured without it refused --restore-processors at every restore. OpenVMM now accepts one maxcpus= token on a platform command line, the canonical form of the recorded boot-online count, as it accepts nr_cpus=, so each clone of one platform capture can bring its own vCPU prefix online at restore. The second stops the control-console broker from timing a host's authentication while host input is gated. A host that connected as soon as it launched a gated restore, and authenticated right away, was detached whenever the guest's repair outlasted --microvm-control-auth-timeout-ms, although the broker had not read its authentication yet. The timeout is now suspended while input is gated and restarts when the input is released. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the two commits that the previous commit pins as positions 339 and 340 of the microVM chain, and move the NVX pin to 2b681564d. Position 339 lets platform snapshots opt into restore-time vCPU activation. It fixes position 120, "openvmm: bring a restore-time processor prefix online", which added the activation without extending the platform allowlist that position 116 had introduced, so it folds into 120 in PR20. Position 340 times host authentication only while the control-console broker reads host input. It fixes position 136, "virtio_console: attach the broker to serial I/O with receive credits", so it folds into 136 in PR23. Extend the scope, the chain, the fold instructions, the table of the commits after the mapped tip, the PR summary, and the commit mapping, and count the fixups in the PR20 and PR23 nodes of the graph. The pin is now 183 commits past the mapped tip. Position 338, 8a0d945ed, is no longer only a proposal: microsoft/openvmm#4734 merged it, so it is the head of feature/microvm/dev, one commit on 79a501b63, which remains the head of feature/microvm/release/v2.3. Say so in the scope and in position 338's entry of the commit mapping, and name microsoft/openvmm#4735 in the scope as the proposal of positions 339 and 340 for feature/microvm/dev. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The pinned OpenVMM accepts restore-time vCPU activation on platform snapshots. Say in the run guide's restore-readiness section that a platform snapshot opts in as other snapshots do, by capturing with a canonical maxcpus= boot-online prefix, so each clone of one platform capture can bring its own prefix online. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No test covered restore-time vCPU activation on a platform snapshot, which OpenVMM refused until the pinned fix. Add the platform-restore-processors microVM scenario to the default suite. The scenario captures a platform snapshot of capacity 4 whose only caller token is maxcpus=1, with a distro layer and a scratch disk, so the guest's nvx-snapshot takes the platform tier from the host. It restores that snapshot at each requested --restore-processors count up to the capacity, 1, 2, and 4 by default, and once without a target, each with a fresh scratch disk. Each restore must run the platform repair and pass the checks of restore-processors: the online CPU prefix, the warp probe over every pair of online CPUs, the time ABI's restore line, and the VP-binding profile, which on MSHV must bind only the target's prefix. OpenVMM must refuse the counts 0 and 3, which no microVM supports, and 8, above the capacity, before the guest runs, and no restore may change the snapshot. The scenario needs sandbox blocks, so it runs only with the Alpine sandbox-control guest, like snapshot-tiers. It took 10 to 17 s on bare-metal KVM, MSHV, and WHP hosts, so the shards are planned again with an estimate of 16 s, which CI's duration reports replace once CI has run it. Unit tests cover the capture and restore commands, the refused counts, the platform repair check, the MSHV prefix check, and the snapshot's integrity, and the runner tests cover its processor counts and its duration record. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the platform-snapshot-restore-vcpu workload suite, which restores one platform snapshot of capacity 8 at the online targets 1, 2, 4, and 8. Like snapshot-restore-vcpu, it reports each target's restore latency and OpenVMM's peak RSS, and its lifecycle phases with --snapshot-profile. A platform command line carries no tenant tokens, so the capture boots with maxcpus=1 as its only caller token, a distro layer, and a scratch disk, and the guest captures from its shell instead of with the pre-banner capture. Each restore reattaches the blocks and reaches its marker after the guest's platform repair, which onlines the target's prefix, refreshes the clone's identity, and acknowledges the restore gate. The restored guest then waits for the host to end it: a sandbox guest's virtio console can deliver the marker after a prequeued exit has started OpenVMM's teardown, too late to sample its peak RSS. The suite is explicit, so it changes no performance baseline. On bare-metal hosts, its medians are close to snapshot-restore-vcpu's at every target: for 1, 2, 4, and 8 vCPUs, 82, 136, 154, and 232 ms on Linux/MSHV, 68, 152, 200, and 231 ms on Linux/KVM, and 178, 385, 486, and 530 ms on Windows/WHP. Part of #796 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/fix-issue-796-implement-tests-benchmarks
branch
from
October 25, 2026 12:06
f593442 to
42bfb8e
Compare
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-issue-796-implement-tests-benchmarks
branch
October 25, 2026 13:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #796. Pins microsoft/openvmm#4735 at
2b681564dd63483f2f9c9f9bba3b98a4c00e9680, which lets platform snapshots bring a different vCPU prefix online in each clone at restore, as they can already grow their RAM. It adds a microVM scenario and a benchmark that cover that gap.Based on
devatf4494453f. #797 and microsoft/openvmm#4734 have merged, so this branch's five commits are its whole diff. microsoft/openvmm#4735 is two commits directly on8a0d945ed, which is now the head offeature/microvm/devand the revision thatdevpins.Change
30af3d6f8openvmmopenvmm8a0d945ed→2b681564d: the two commits of microsoft/openvmm#473524970cdeddocdoc/openvmm-upstream-roadmap.mdfeature/microvm/devb5899557cdocdoc/run.md--restore-processorsafter amaxcpus=capturea6971a8bascriptsscripts/nvx_tools/microvm_tests.py,scripts/test_microvm_tests.py,scripts/nvx_tools/microvm_shards.json,doc/ci.md,doc/usage.mdplatform-restore-processors; shards replanned42bfb8e13scriptsscripts/nvx_tools/benchmark.py,scripts/test_nvx_tools.py,doc/benchmarks.md,doc/usage.mdplatform-snapshot-restore-vcpuOpenVMM fixes
maxcpus=(2f51e8bab). The platform tier's command-line allowlist acceptednr_cpus=but notmaxcpus=. A platform capture booted withmaxcpus=Btherefore failed to publish withplatform snapshot command line contains tenant or unsupported configuration, and one booted without it refused--restore-processors. OpenVMM now accepts onemaxcpus=token on a platform command line: the canonical form of the recorded boot-online count, supported and within the capacity.maxcpus=01,maxcpus=+1, duplicates, and mismatches stay rejected.2b681564d), the issue's lower-priority item. The broker enforced the authentication deadline even while a gated restore kept it from reading the host's authentication, so a host that connected at launch was detached whenever repair outlasted--microvm-control-auth-timeout-ms. The deadline is now suspended while input is gated and restarts at the release for a host that has not authenticated.Tests and benchmark
platform-restore-processorscaptures a platform snapshot of capacity 4 whose only caller token ismaxcpus=1, with sandbox blocks. It restores the snapshot at each requested count up to 4 (1, 2, and 4 by default) and once without a target, each with fresh scratch. Each restore must run the platform repair and pass the checks ofrestore-processors: the online CPU prefix, the warp probe, the time ABI's restore line, and the VP bindings (MSHV binds only the target's prefix). OpenVMM must refuse 0, 3, and 8 before the guest runs, and the snapshot must not change. It runs with the Alpine sandbox-control guest only, likesnapshot-tiers, and joins shard 1 on every backend with a 16 s estimate.platform-snapshot-restore-vcpurestores one capacity-8 platform snapshot at the online targets 1, 2, 4, and 8 and reports latency, peak RSS, and, with--snapshot-profile, the lifecycle phases. It is explicit, so no performance baseline changes.Evidence
On the bare-metal hosts prometheus30 (Linux/MSHV), prometheus32 (Linux/KVM), and prometheus28 (Windows/WHP), each in a separate worktree:
79a501b63binaryplatform snapshot command line contains tenant or unsupported configuratione2eacceptance benchmarkRestore medians in ms at 1, 2, 4, and 8 online vCPUs (capacity 8, 512 MiB, one warmup, five runs):
platform-snapshot-restore-vcpusnapshot-restore-vcpuEquivalence and validation
f593442be, whose CI passed (run 40018285211), from openvmm: write each page of a new microVM RAM file once at capture #797's headaaf30ed88ontodevatf4494453f; all five applied without conflicts.git range-diffshows thedoc/run.mdcommit and bothscriptscommits patch-identical. The gitlink commit changes only its message, which no longer calls8a0d945eda proposal. The roadmap commit also corrects the two statements that [DO NOT REVIEW] microvm: flush writable file views whole, in one ascending pass openvmm#4734's merge made stale: the scope now names8a0d945edas the head offeature/microvm/dev, one commit on79a501b63, which remains the head offeature/microvm/release/v2.3, and position 338's entry says that #4734 merged it.8a0d945ed, the current head offeature/microvm/dev, so rebasing the two commits onto that branch is a no-op, and both stay signed with their reviewed SHAs (treee571d9569). Its protocol vectors are unchanged too: the copies innvx-rs/tests/vectors/openvmm/match the submodule's files at2b681564d(cargo test --test openvmm_vectors -- --ignored).79a501b63and these NVX changes on openvmm: pin microVM v2.3, rebased onto upstream main #788's head. That is patch equivalence with this branch, not tree identity: the OpenVMM diff has the same patch-id (174b483ff) on79a501b63and on8a0d945ed, and the NVX diff, without the gitlink and the roadmap, has the same patch-id (2b42f7345) onee2e3c7e5, onaaf30ed88, and onf4494453f.cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agentforopenvmm_helpers,virtio_console, andopenvmm_entry(404 passed), andcargo xtask fmt(clean). OpenVMM's CI passes every job butrun vmm-tests [x64-linux-amd-snp-mshv], whose two SNP boot tests fail in 0.05 s, before any guest code runs, at partition creation withfailed to initialize VM: Kernel returned errno: I/O error (os error 5)on the runnermshv-snp-4. [DO NOT REVIEW] microvm: flush writable file views whole, in one ascending pass openvmm#4734's run passed that job onmshv-snp-2, and neither commit touches partition creation, so the failed jobs are being re-run.nvx.py verify,ruff check,ruff format --check,pyrightfor Linux and Windows, the markdown link check, thevalidate-nvxunittest list excepttest_setup_azure.py(1,270 tests, 53 skipped),test_adversarial.py,test_hosts.py, and the CLI--helpchecks.test_setup_azure.py, which need a Linuxbash, and the Specula wiring tests, which import the Linux-onlyfcntl;shellcheckandshfmt, which run in Docker, although no shell script changed; the microVM scenarios and the benchmark; and the OpenVMM VMM tests. CI runs all of them.Exact NVX head submitted to CI:
42bfb8e13d63b4e04b15bb67b98f0044b3dd9259.